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Primary  goals  of  health  surveillance  in  the  military  include  monitoring  the  health  status 
of  military  personnel,  and  detecting  outbreaks  of  naturally  occurring  and  bioterrorism-related 
epidemics.  Two  near  real-time  automated  surveillance  systems  currently  in  use  by  the 
Department  of  Defense  (DoD)  are  the  Medical  Data  Surveillance  System  (MDSS),  and  the 
Electronic  Surveillance  System  for  the  Early  Notification  of  Community-Based  Epidemics 
(ESSENCE). 

Both  prototype  systems  are  “passive”  surveillance  systems  in  that  they  use  already 
collected  diagnostic  data  reported  as  primary  International  Classification  of  Diseases,  9th 
Revision  (ICD-9)  codes  on  automated  medical  encounter  records.  The  advantages  of  passive 
surveillance  systems  are  many.  These  systems  can  rapidly  detect  unusual  disease  events,  and  a 
large  number  of  individuals  can  be  placed  under  surveillance.  Passive  systems  impose  no 
additional  reporting  burden  on  health  care  providers,  and  avoid  the  reporting  bias  inherent  in 
non-automated  surveillance  systems.  In  contrast,  traditional  active  surveillance  (i.e.,  “hot 
pursuit”)  identifies  cases  by  active  searching  and  regular  case  reports  from  health  care  facilities. 
Although  more  expensive  than  passive  surveillance,  active  surveillance  data  are  usually 
considered  more  complete  and  accurate  (Lilienfeld  &  Stolley,  1994). 

Although  MDSS  and  ESSENCE  use  the  same  source  of  data  (i.e.,  primary  ICD-9  codes 
entered  into  the  Ambulatory  Data  System),  they  differ  in  their  disease  measures  (i.e.,  reports), 
their  assignment  of  ICD-9  codes  into  disease/syndrome  categories,  and  their  outbreak  detection 
algorithms.  Studies  evaluating  the  impact  of  these  methodological  differences  are  few.  Our 
previous  research  reported  substantial  differences  between  MDSS  Version  3.1  and  ESSENCE  IB 
in  their  measures  of  respiratory  illness,  the  number  of  ICD-9  codes  used,  the  resulting  number  of 
cases  captured,  and  leading  diagnoses  (Howe,  Faix,  Woodruff,  &  Kleiner,  2004).  Interestingly, 
the  differences  did  not  appear  to  substantially  alter  conclusions  about  patient  characteristics  or 
at-risk  groups.  One  unpublished  retrospective  study  compared  MDSS  and  ESSENCE  detection 
of  a  Streptococcus  A  outbreak  at  San  Diego  clinics  in  2002  (Melcer,  2003).  Both  systems 
generated  outbreak  alerts,  although  MDSS  alerted  earlier  and  was  found  to  be  more  sensitive  to 
changes.  Another  study  examined  the  concordance  between  broad  respiratory-related  syndromic 
categories  in  ESSENCE  and  active  surveillance  of  febrile  respiratory  illness  (FRI)  at  9  medical 
treatment  facilities  (Marsden-Haug,  Elbert,  Hawksworth,  Foster,  &  Pavlin,  2002).  ESSENCE 
was  able  to  approximate  the  same  FRI  trends  as  the  active  surveillance  system  at  several  of  the 
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facilities;  however,  as  might  be  expected,  ESSENCE  counts  were  much  higher  than  those 
identified  by  FRI-specific  active  surveillance. 

The  purpose  of  this  paper  was  to  add  to  the  limited  information  about  surveillance  data 
using  different  mechanisms  by  retrospectively  comparing  the  two  systems’  surveillance  trends 
and  potential  outbreak  detection  of  respiratory  illness  at  the  Marine  Corps  Recruit  Depot 
(MCRD)  San  Diego.  The  choice  of  respiratory  illness  at  MCRD  for  comparison  is  more  than  an 
academic  one.  The  DoD  is  concerned  about  respiratory  illness,  including  FRI,  for  several 
reasons.  U.S.  military  populations  have  been  subject  to  respiratory  disease  epidemics  in  the  past, 
most  notably  during  the  influenza  pandemic  of  1918,  when  43,000  U.S.  troops  died  from  the 
illness.  Military  personnel  are  deployed  worldwide,  often  in  areas  that  have  a  high  prevalence  of 
respiratory  illness  such  as  influenza.  Also,  the  military  population  is  highly  mobile,  providing 
opportunities  to  spread  illness  to  a  large  number  of  people  over  a  wide  area.  Respiratory  illnesses 
have  long  been  a  problem  particularly  in  recruit  training  camps,  where  close  quarters  facilitate 
rapid  transmission. 

In  addition  to  the  two  passive  automated  systems,  data  were  included  from  a  third 
distinctly  different  type  of  surveillance  system — traditional  active  surveillance  of  FRI  at  MCRD. 
Because  all  three  systems  have  different  case  definitions  and  produce  different  disease  measures, 
the  systems  were  compared  in  a  general  way,  focusing  on  patterns  of  change  over  time.  Of 
primary  interest  was  the  comparison  of  the  systems’  respiratory  illness  trends  and  outbreak 
detection.  In  addition,  we  assessed  the  extent  to  which  a  passive  system  could  capture  the  same 
trend  as  a  traditional,  active  surveillance  effort  specifically  monitoring  FRI  at  MCRD. 

Methods 

Respiratory-related  diagnoses  were  obtained  retrospectively  from  MDSS  Version  3.1, 
ESSENCE  IV,  and  active  FRI  surveillance  for  the  64-week  period  of  July  29,  2002,  to  October 
19,  2003.  Counts  were  used  rather  than  rates  because  ESSENCE  does  not  calculate  rates.  Weekly 
counts  rather  daily  counts  were  compared  because  active  FRI  surveillance  data  are  reported  on  a 
weekly  rather  than  daily  basis.  All  three  systems  (particularly  MDSS  and  ESSENCE)  may 
include  follow-up  visits  in  their  weekly  counts  (our  previous  work  has  indicated  that  follow-up 
visits  in  the  recruit  population  may  be  as  high  as  50%).  MDSS  and  ESSENCE  can  distinguish 
between  new  and  follow-up  cases;  however,  their  definitions  of  an  initial  versus  follow-up  visit 
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are  quite  different.  Further,  active  FRI  surveillance  reporting  does  not  allow  for  identifying  new 
versus  follow-up  visits.  For  this  reason,  total  counts  are  used  from  all  systems.  Occasionally  a 
nonrecruit  may  be  treated  at  the  MCRD  clinic,  and  MDSS  and  ESSENCE  will  include  these 
cases.  However,  these  cases  are  few,  and  because  our  primary  question  is  how  well  the  systems 
can  easily  reflect  respiratory  illness  without  additional  analysis  for  selecting  certain  cases,  data 
were  used  as  is  without  removing  nonrecruits. 

MDSS  and  ESSENCE  each  have  multiple  respiratory-related  automated  reports.  Three 
reports  from  MDSS  and  two  from  ESSENCE  were  included  (see  below),  in  addition  to  results 
from  active  FRI  surveillance. 

Description  of  MDSS 

MDSS,  developed  by  the  Naval  Health  Research  Center  (NHRC)  and  the  Space  and 
Naval  Warfare  Systems  Command  San  Diego,  is  a  Web-based  medical  surveillance  system  that 
provides  near  real-time  medical  threat  assessment  using  routinely  collected  medical  data.  MDSS 
is  currently  being  used  to  monitor  illness  trends  and  potential  epidemics  at  medical  treatment 
facilities  (MTFs)  (hospitals  and  clinics)  in  the  San  Diego  area.  Data  come  from  the  DoD 
Standardized  Ambulatory  Data  Record  (SADR),  records  that  are  designed  primarily  for  health 
care  management  and  cost  monitoring.  For  every  outpatient  encounter  within  the  DoD,  a  SADR 
is  produced  and  coordinated  with  some  patient  demographic  data  on  a  daily  basis.  Data  include 
visits  by  active-duty  personnel  and  their  families  as  well  as  retirees  and  other  non-active-duty 
beneficiaries.  The  SADR  is  usually  completed  at  the  place  and  time  of  the  patient  encounter.  The 
provider  completes  the  SADR  with  a  known  or  presumed  diagnosis  from  the  ICD-9.  Within 
MDSS,  ICD-9  codes  are  grouped  into  several  broader  reports,  including  the  Disease  and  Non- 
Battle  Injury  (DNBI)  report.  DNBI  categories  include  Total  Respiratory  visits,  as  well  as  Lower 
and  Upper  Respiratory  visits.  MDSS  provides  counts  as  well  as  rates  for  each  of  the  MTFs. 

Among  its  monitoring  capabilities,  MDSS  applies  a  Dynamic  Change-point  Detection 
(DCD)  algorithm  to  quickly  detect  significant  increases  in  disease  trends.  The  DCD  employs  the 
Shewhart  test,  as  well  as  an  extended  cumulative  sum  (CUSUM)  method,  methods  commonly 
used  in  industry  to  detect  unwanted  changes  in  industrial  processes.  CUSUM  methods  operate  by 
accumulating  deviations  between  observations  and  expectations.  When  cumulated  deviations 
exceed  some  predefined  threshold,  an  alarm  or  alert  is  generated,  indicating  an  increase  in  the 
mean  of  the  underlying  condition.  DCD  can  detect  different  degrees  of  outliers  (i.e.,  shifts, 
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trends).  In  the  present  study,  we  focused  on  the  most  extreme  statistical  event  (i.e.,  burst),  a  data 
point  3  or  more  standard  deviations  away  from  the  mean.  The  MDSS  Reference  Manual  provides 
more  detail  and  a  mathematical  description  of  the  DCD  algorithm  (Pugh,  2003). 

Description  of  ESSENCE 

ESSENCE,  developed  by  the  DoD  Global  Emerging  Infections  System  at  Walter  Reed 
Army  Institute  of  Research,  was  designed  for  early  discovery  of  infectious  disease  outbreaks  at 
MTFs  and  in  the  civilian  sector.  Like  MDSS,  ESSENCE  pulls  SADR  outpatient  data  on  a  daily 
basis,  and  categorizes  ICD-9  codes  into  broader  groupings.  While  MDSS  pulls  SADR  data 
directly  from  servers  at  local  MTFs,  ESSENCE  is  a  DoD-wide  system  that  pulls  data  from  a 
central  TRICARE  server.  Seven  preset  ESSENCE  syndrome  groupings  focus  on  infectious 
disease.  Two  ESSENCE  syndrome  groupings  are  included  in  this  study.  The  first  is  the 
ESSENCE  Respiratory  Syndrome  that  includes  specific  and  nonspecific  diagnoses  and 
symptoms  of  acute  respiratory  tract  infections,  while  excluding  chronic  conditions  such  as 
bronchitis  and  asthma.  The  second  ESSENCE  syndrome  included  is  the  Febrile  Illness 
Syndrome,  a  fairly  broad  category  that  captures  febrile  conditions  with  both  unknown  and 
specific  causes. 

Two  other  ESSENCE  categories,  DNBI  Total  Respiratory  and  Influenza-like  Illness 
(ILI),  were  considered  for  inclusion  in  the  comparison,  but  were  not  used.  ESSENCE  DNBI 
Total  Respiratory  was  so  highly  correlated  with  ESSENCE  Respiratory  Syndrome  (r  =  .99),  it 
was  considered  redundant.  Also,  ESSENCE  detection  algorithms  are  not  run  on  the  DNBI  data. 
ILI  within  ESSENCE  is  measured  as  a  percentage  of  total  visits  (versus  actual  counts)  and  is 
presented  for  all  San  Diego  MTFs  combined;  therefore,  it  was  not  included  in  this  comparison  of 
MCRD-specific  respiratory  conditions. 

For  detecting  potential  outbreaks  in  ESSENCE  IV,  a  prediction  of  normal  daily  ranges 
for  the  syndrome  group  is  done  using  historical  data.  Regression  models  base  daily  predictions 
on  an  empirically  derived  baseline  period  covering  between  4  and  12  weeks  of  ESSENCE  data. 
Confidence  intervals  are  fitted  around  predicted  daily  counts,  and  the  upper  bound  is  used  as  a 
threshold.  Numbers  of  daily  patient  encounters  that  fall  outside  the  confidence  interval  indicate  a 
departure  from  the  values  that  are  expected.  If  the  observed  count  exceeds  the  99%  confidence 
interval,  it  is  considered  to  be  a  high-level  warning.  (See 

https://essencewww.ihuapl.edu/isipp/servlet/HomePageServlet  for  more  in-depth  information 
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about  ESSENCE  and  its  outbreak  detection  algorithms.  A  user  ID  and  password  must  first  be 
acquired  and  can  be  requested  at  Essence@deplovmenthealth.osd.min. 

Description  of  Active  FRI  Surveillance 

Active  FRI  surveillance  is  ongoing  at  MCRD  San  Diego  and  7  other  U.S.  military 
training  sites  to  provide  an  early  warning  system  for  detection  of  potential  FRI  epidemics  in  the 
recruit  population.  Rather  than  identifying  cases  from  ICD-9  codes  in  automated  medical 
records,  trained  personnel  at  MCRD  actively  monitor  their  training  population  for  FRI.  Active- 
duty  trainees  who  visit  the  clinic  with  an  oral  temperature  of  100.5°F  or  higher,  and  who  have  a 
respiratory  symptom  such  as  a  cough  or  sore  throat,  are  counted  as  FRI  cases.  In  addition,  any 
trainee  diagnosed  with  pneumonia  is  considered  a  case  of  FRI.  MCRD  reports  its  FRI  counts  and 
population  numbers  to  investigators  at  the  NHRC  on  a  weekly  basis,  who  then  publish  weekly 
FRI  rates  on  the  Internet  for  rapid  dissemination  of  information. 

For  detection  of  potential  outbreaks,  NHRC  classifies  the  MCRD  weekly  FRI  rate  into 
one  of  three  status  categories:  (a)  at  or  below  an  expected  value,  (b)  moderately  elevated,  or  (c) 
substantially  elevated.  A  combination  of  fixed  and  variable  alarm  thresholds  are  used  to 
determine  FRI  rate  status.  Fixed  thresholds  are  generated  based  on  the  90th  and  95th  percentiles 
of  their  past  rates.  Variable  thresholds  are  calculated  using  autoregressive  integrated  moving 
average  (ARIMA)  modeling.  In  the  case  of  differences  in  the  fixed  and  variable  status,  the  more 
extreme  FRI  rate  status  is  used  for  that  week’s  classification.  More  information  about  FRI 
surveillance  at  MCRD  can  be  found  at  http://www.nhrc.navy.mil/geis/. 

Analysis 

Analyses  are  descriptive  and  correlational.  Averages  (means  or  medians)  are  used  to 
describe  characteristics  and  surveillance  results  from  the  three  systems’  various  reports, 
including  the  average  weekly  number  of  cases  captured.  Weekly  counts  of  respiratory-related 
illness  and  system  outbreak  alerts  are  presented  graphically.  Pearson  correlations  are  used  to 
examine  the  associations  between  the  three  systems’  counts  and  alerts. 
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Results 


ICD-9  Coding  Differences  in  MDSS  and  ESSENCE 

Table  1  presents  the  number  of  ICD-9  codes  used  by  MDSS  and  ESSENCE  in  their 
various  respiratory-related  reports.  While  ESSENCE  codes  focus  on  infectious  illnesses,  MDSS 
coding  reflects  both  infectious  and  chronic  respiratory  conditions.  MDSS  uses  more  ICD-9  codes 
for  its  Total  Respiratory  report  than  the  broadest  ESSENCE  report,  Respiratory  Syndrome  (209 
vs.  137  ICD-9  codes).  ESSENCE  Respiratory  Syndrome  report  typically  categorized  more  cases 
than  MDSS  (mean  of  about  180  cases  per  week  vs.  160  cases  per  week;  median  of  127  vs.  88 
cases  per  week).  The  ESSENCE  Febrile  Illness  Syndrome  report  had  few  cases  over  the 
surveillance  period,  with  a  median  weekly  number  of  0  (mean  =  2.6).  Cases  reported  by  active 
FRI  surveillance  was  relatively  low,  with  a  median  of  13  cases  per  week  (mean  =  16).  Typically, 
MDSS  and  ESSENCE  had  the  same  leading  diagnoses:  465.9  (Acute  upper  respiratory  infections 
of  multiple  or  unspecified  sites),  466.0  (Acute  bronchitis  or  bronchiolitis),  and  486  (Pneumonia, 
organism  unspecified). 

Comparison  of  Respiratory  Illness  Trends 

Figure  1  presents  the  weekly  number  of  cases  reported  by  each  surveillance  system  over 
the  64-week  period.  As  described  in  the  Methods,  3  reports  are  shown  from  MDSS-DNBI  Total 
Respiratory,  DNBI  Upper  Respiratory,  and  DNBI  Lower  Respiratory.  Due  to  technical 
difficulties,  MDSS  failed  to  collect  data  for  the  period  of  September  30  through  October  13, 
2002,  therefore  no  data  are  reported  from  MDSS  for  those  2  weeks.  Two  reports  from  ESSENCE 
are  presented — Respiratory  Infectious  Disease  Syndrome  and  Febrile  Illness  Syndrome.  The  thin 
solid  line  represents  cases  identified  through  active  FRI  surveillance  at  MCRD. 

ESSENCE  Respiratory  Syndrome  usually  reported  the  most  cases  per  week,  followed 
closely  by  cases  reported  by  MDSS  Total  Respiratory.  ESSENCE  Respiratory  Syndrome,  MDSS 
Total  Respiratory,  and  MDSS  Upper  Respiratory  counts  showed  a  similar  pattern  of  increases 
and  decreases  week  to  week.  Not  surprising,  trends  in  the  number  of  cases  appearing  in  the 
MDSS  Lower  Respiratory  report  were  somewhat  independent  of  MDSS  Total  and  Upper 
Respiratory  trends.  As  one  would  expect,  the  numbers  of  FRI  cases  identified  through  active 
surveillance  were  relatively  low,  ranging  from  1  to  50  per  week  (median  of  13  and  mean  of  15.5 
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cases  per  week).  The  number  of  cases  from  the  ESSENCE  Febrile  Illness  Syndrome  report  also 
was  low,  ranging  from  0  to  42  cases  per  week,  with  over  half  of  the  weeks  showing  0  cases. 

Table  2  presents  Pearson  correlations  for  the  systems’  various  reports.  Weekly  counts 
captured  by  the  MDSS  Total  Respiratory  report  were  very  highly  correlated  with  ESSENCE’S 
Respiratory  Syndrome  ( r  =  .985).  MDSS  Upper  Respiratory  and  Lower  Respiratory  counts  also 
were  highly  correlated  with  ESSENCE  Respiratory  Syndrome  (.81 -.90).  ESSENCE  Febrile 
Illness  Syndrome  was  not  strongly  related  to  any  other  report,  including  active  FRI  surveillance. 
Active  FRI  surveillance  was  most  strongly  associated  with  MDSS  Upper  Respiratory  (.62), 
closely  followed  by  ESSENCE  Respiratory  Syndrome  (.57),  and  MDSS  Total  Respiratory  (.56). 

Potential  Outbreak  Detection 

Although  MDSS  and  ESSENCE  collect  data  and  issue  alerts  of  significantly 
increasing  trends  on  a  daily  basis,  active  FRI  surveillance  at  MCRD  aggregates  data  and  issues 
alerts  based  on  weekly  data.  Therefore,  alert  information  is  presented  on  a  weekly  basis  for 
comparison  purposes.  Table  1  presents  the  number  of  weeks  during  the  64-week  surveillance 
period  that  each  system  generated  at  least  one  alert.  Figure  2  shows  the  information  graphically 
by  week  with  symbols  corresponding  with  the  patient  visit  counts.  The  MDSS  Total  Respiratory 
report  produced  at  least  one  alert  in  39  of  the  64  weeks  (61%  of  all  weeks);  MDSS  Upper  and 
Lower  Respiratory  reports  produced  even  more  alerts  (63-65%  of  weeks).  These  numbers  are  in 
sharp  contrast  to  the  1 1  alerts  issued  by  ESSENCE  Respiratory  Syndrome  (17%  of  all  weeks). 
The  ESSENCE  Febrile  Illness  Syndrome  alerted  only  3  of  the  64  weeks.  Active  FRI  surveillance 
at  MCRD  produced  alerts  for  22  of  the  64  weeks  (34%),  with  most  of  the  alerts  occurring  in  the 
final  weeks  of  the  surveillance  period.  Pearson  and  Spearman  correlation  coefficients  computed 
for  (a)  the  number  of  alerts  per  week,  and  (b)  the  presence  (yes  vs.  no)  of  any  alert  during  the 
week  confirmed  relatively  low  associations  among  MDSS,  ESSENCE,  and  active  FRI 
surveillance  systems  (r  =  .05-. 28). 

Considering  number  of  alerts  on  a  daily  basis  for  MDSS  and  ESSENCE,  MDSS  produced 
many  more  alerts  than  did  ESSENCE  (see  Table  1).  About  20%  ( n  =  80-90)  of  the  448  days  of 
the  surveillance  period  showed  an  alert  by  one  or  more  of  the  MDSS  various  respiratory  reports, 
compared  with  3%  (14  days)  by  the  ESSENCE  Respiratory  Syndrome  report. 

A  known  outbreak  of  Streptococcus  A  the  second  week  of  December  2002  was  used  to 
assess  if  and  how  rapidly  the  systems  alerted.  Active  FRI  surveillance  generated  an  alert  for  that 


week,  but  because  active  surveillance  results  are  only  reported  weekly,  daily  data  are  not 
available.  MDSS  and  ESSENCE,  on  the  other  hand,  gather  data  and  create  reports  on  a  daily 
basis;  therefore,  they  were  compared  to  assess  how  quickly  each  alerted  that  week.  The  MDSS 
Total  Respiratory  report  produced  an  alert  on  December  11,  with  many  of  the  cases  diagnosed  as 
possible  Streptococcus  A  infection.  ESSENCE’S  Respiratory  Syndrome  alerted  on  December  14. 

Discussion 

Despite  differences  in  ICD-9  code  categorization  methods  and  resulting  disease 
measures,  automated  reports  generated  by  MDSS  3.1  and  ESSENCE  IV  showed  similar  trends  in 
respiratory  illness  at  MCRD  San  Diego  over  a  64-week  period.  In  particular,  the  systems’ 
broadest  respiratory  reports,  MDSS  Total  Respiratory  and  ESSENCE  Respiratory  Syndrome, 
showed  similarity  in  the  numbers  of  respiratory  cases,  and  changes  over  time.  There  was  a  very 
high  correlation  between  these  two  reports  (.98),  although  there  was  variability  in  the 
correlations  among  some  of  the  other  automated  reports.  For  example,  ESSENCE  Febrile  Illness 
Syndrome  showed  very  low  counts  and  low  correlations  with  other  automated  respiratory 
measures,  including  FRI  counts  captured  through  active  surveillance.  It  may  be  that  the 
particular  ICD-9  codes  ESSENCE  uses  for  its  Febrile  Illness  Syndrome  are  either  rare  conditions 
(e.g.,  Plague-020.0),  or  rarely  assigned  as  a  primary  diagnosis  (e.g.,  Fever-780.6). 

The  MDSS  Total  Respiratory  and  Upper  Respiratory  reports  and  ESSENCE’S 
Respiratory  Syndrome  were  also  somewhat  successful  in  reflecting  respiratory  illness  trends 
from  a  distinctly  different  type  of  surveillance  system — traditional  active  surveillance  of  FRI. 

Not  surprising,  MDSS  and  ESSENCE  counts  of  respiratory  illness  were  much  higher  than  FRI 
counts  from  active  surveillance,  most  likely  because  their  respiratory  categories  use  a  relatively 
large  number  of  ICD-9  codes,  many  of  which  are  not  related  to  FRI.  Despite  that,  correlations 
between  active  FRI  surveillance  and  the  automated  diagnoses-based  systems  ranged  from  .56- 
.62,  indicating  fairly  high  correlations  of  their  counts. 

While  counts  were  relatively  highly  correlated  among  the  systems  (depending  upon  the 
report),  number  and  presence  of  alerts  generated  by  the  systems  were  not  highly  related.  The 
broadest  MDSS  report  alerted  about  61%  of  all  weeks,  ESSENCE  alerted  17%  of  all  weeks,  and 
active  FRI  surveillance  was  intermediate,  alerting  34%  of  all  weeks.  Although  ESSENCE  was 
much  more  parsimonious  in  its  outbreak  alerts,  MDSS  indicated  a  later-confirmed  outbreak  a 
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few  days  earlier  than  did  ESSENCE.  Users’  reports  of  MDSS  sensitivity  has  been  has  been 
reported  elsewhere  (Melcer,  2003).  While  some  view  the  relatively  high  sensitivity  to  change  of 
MDSS  as  a  strength,  others  consider  it  a  drawback  because  of  the  resources  and  time  needed  to 
further  investigate  each  alert,  and  because  of  the  likelihood  of  false  alarms.  The  optimal  level  of 
sensitivity  of  a  system  probably  differs  depending  on  the  needs  and  interest  of  the  user. 

Limitations 

One  limitation  in  this  comparison  study  was  the  inability  to  match  individual  patient 
records  across  the  systems  to  determine  the  extent  to  which  the  same  individuals  were 
“captured.”  Although  the  three  systems  use  overlapping  patient  populations  (i.e.,  outpatient  visits 
at  the  MCRD  clinic),  a  common  identifier  did  not  exist.  In  addition,  patient  population 
characteristics  (e.g.,  age,  sex)  are  not  readily  available  for  all  three  systems;  therefore,  it  was  not 
possible  to  confirm  the  similarity  of  the  patient  populations.  Furthermore,  there  are  likely 
inaccuracies  in  the  data  (e.g.,  underreporting  of  cases  from  active  FRI  surveillance, 
nonstandardized  ICD-9  coding  in  MDSS  and  ESSENCE),  the  extent  of  which  is  not  known.  A 
comparison  of  initial  cases  (rather  than  total)  would  have  been  valuable,  but  was  not  done 
because  of  the  systems’  inconsistent  definitions  of  incidence.  Finally,  generalizability  of  these 
results  to  other  populations  and  other  medical  conditions  cannot  be  assumed. 

Differences  in  disease  measures  make  comparisons  between  the  two  automated 
surveillance  systems  and  active  surveillance  results  difficult.  There  is  no  system  or  measure  in 
the  present  study  that  could  be  considered  a  gold  standard;  therefore,  we  cannot  recommend  one 
system,  or  report  within  a  system,  as  superior.  In  fact,  all  three  systems  included  seem  to  each 
have  their  specific  strengths  and  shortcomings. 

Conclusions 

MDSS  and  ESSENCE  measures,  particularly  their  broadest  respiratory-related  reports, 
appear  to  be  useful  tools  for  surveillance  of  respiratory  illness  in  recruits.  The  basis  of  these  two 
systems,  ICD-9  codes,  often  reflect  unconfirmed  diagnoses  and  nonspecific  symptoms. 
Therefore,  passive  surveillance  systems  will  not  replace  more  traditional  surveillance  and  public 
health  reporting  systems.  However,  because  these  systems  are  far  less  resource  intensive,  and 
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reasonably  reflect  disease  trends,  they  may  be  valuable  adjuncts  for  surveillance  of  other  medical 
conditions  and  populations. 
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Table  1.  Number  of  ICD-9  Codes  Used,  Average  Weekly  Number  of  Cases,  and  Number  of 
Alerts  Issued,  by  Surveillance  System  —  MCRD  San  Diego,  29  July  2002-19  October  2003 


Surveillance  System/Report 


MDSS 

Total 

Resp 

MDSS 

Upper 

Resp 

MDSS 

Lower 

Resp 

ESSENCE 

Resp 

Synd 

ESSENCE 
Febrile  Ill. 
Synd 

Active 

FRI 

Surveill 

No.  of  ICD-9  codes 
used  by  system 

209 

81 

128 

137 

114 

N/A 

Number  of  weekly 
cases  (median) 

87.5 

62.5 

43.7 

126.5 

0.0 

13.0 

Number  of  weekly 
cases  (mean) 

159.9 

116.2 

19.0 

180.2 

2.6 

15.5 

Number  of  weeks 
with  at  least  one 
alert 

39 

42 

40 

11 

3 

22 

Days  alerted 

90 

80 

90 

14 

3 

N/A 

13 


Table  2.  Pearson  Correlation  Coefficients  for  Surveillance  Systems’  Weekly  Counts  of 
Respiratory-related  Illness  —  MCRD  San  Diego,  29  July  2002-19  October  2003 


Surveillance  System/Report 


MDSS 

MDSS 

MDSS 

ESSENCE 

ESSENCE 

Active 

Total 

Upper 

Lower 

Resp 

Febrile  Ill. 

FRI 

Resp 

Resp 

Resp 

Synd 

Synd 

Surveill 

MDSS 

Total  Resp 

1.00 

MDSS 

Upper  Resp 

.895** 

1.00 

MDSS 

Lower  Resp 

.806** 

.458** 

1.00 

ESSENCE 

Resp  Synd 

.985** 

.864** 

.818** 

1.00 

ESSENCE 
Febrile  Ill. 

Synd 

.097 

.095 

.067 

.139 

1.00 

Active  FRI 

Surveillance 

— * — : - : — : : : : — 

** 

.561 

** 

.623 

* 

.291 

.569 

.197 

1.00 

Correlation  is  significant  at  the  .05  level  (2-tailed). 
Correlation  is  significant  at  the  .01  level  (2-tailed). 
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Cases 


Figure  1.  Counts  of  Respiratory-related  Illness  from  MDSS,  ESSENCE,  and  Active  FRI  Surveillance,  by  week 
-  MCRD  San  Diego,  29  July  2002-19  October  2003 


Note:  MDSS  missing  data  for  2-week  period  of  9/30-10/13,  2002. 


Cases 


Figure  2.  Alerts  of  Increasing  Respiratory-related  Illness  from  MDSS,  ESSENCE,  and  Active  FRI  Surveillance, 
by  week  --  MCRD  San  Diego,  29  July  2002-19  October  2003 
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Note:  MDSS  missing  data  for  2-week  period  of  9/30-10/13,  2002. 
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